Method and system for multiple f-number lens

ABSTRACT

An imaging lens includes one or more lens elements configured to receive and focus light in a first wavelength range reflected off of one or more first objects onto an image plane, and to receive and focus light in a second wavelength range reflected off of one or more second objects onto the image plane. The imaging lens further includes an aperture stop and a filter positioned at the aperture stop. The filter includes a central region and an outer region surrounding the central region. The central region of the filter is characterized by a first transmission band in the first wavelength range and a second transmission band in the second wavelength range. The outer region of the filter is characterized by a third transmission band in the first wavelength range and substantially low transmittance values in the second wavelength range.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication No. 62/420,249, filed on Nov. 10, 2016, the content of whichis incorporated by reference in its entirety.

BACKGROUND OF THE INVENTION

In optical systems, imaging lenses are utilized to collimate light,focus light, and the like. Despite the progress made in the developmentof optical systems, there is a need in the art for improved imaginglenses.

SUMMARY OF THE INVENTION

The present invention relates generally to imaging systems with amultiple f-number lens. According to an embodiment of the presentinvention, an imaging system includes a near infrared (NIR) light sourceconfigured to emit a plurality of NIR light pulses toward one or morefirst objects. A portion of each of the plurality of NIR light pulsesmay be reflected off of the one or more first objects. The imagingsystem further includes an imaging lens. The imaging lens includes oneor more lens elements configured to receive and focus the portion of theeach of the plurality of NIR light pulses reflected off of the one ormore first objects onto an image plane, and to receive and focus visiblelight reflected off of one or more second objects onto the image plane.The imaging lens further includes an aperture stop, and a filterpositioned at the aperture stop. The filter includes a central regionwith a first linear dimension, and an outer region surrounding thecentral region with a second linear dimension greater than the firstlinear dimension. The central region of the filter is characterized by afirst transmission band in an NIR wavelength range and a secondtransmission band in a visible wavelength range. The outer region of thefilter is characterized by a third transmission band in the NIRwavelength range and substantially low transmittance values in thevisible wavelength range. The imaging system further includes an imagesensor positioned at the image plane. The image sensor includes atwo-dimensional array of pixels. The image sensor is configured todetect a two-dimensional intensity image of the one or more secondobjects in the visible wavelength range at an unbinned pixel resolution,and detect a time-of-flight three-dimensional image of the one or morefirst objects in the NIR wavelength range in a binned pixel resolution.

According to another embodiment of the present invention, an imaginglens includes one or more lens elements configured to receive and focuslight in a first wavelength range reflected off of one or more firstobjects onto an image plane, and to receive and focus light in a secondwavelength range reflected off of one or more second objects onto theimage plane. The imaging lens further includes an aperture stop, and afilter positioned at the aperture stop. The filter includes a centralregion with a first linear dimension, and an outer region surroundingthe central region with a second linear dimension greater than the firstlinear dimension. The central region of the filter is characterized by afirst transmission band in the first wavelength range and a secondtransmission band in the second wavelength range. The outer region ofthe filter is characterized by a third transmission band in the firstwavelength range and substantially low transmittance values in thesecond wavelength range.

According to a yet another embodiment of the present invention, a methodof operating an imaging system is provided. The imaging system includesa near infrared (NIR) light source, an imaging lens, and an image sensorpositioned at an image plane of the imaging lens. The method includesperforming three-dimensional sensing using the imaging system by:emitting, using the NIR light source, a plurality of NIR light pulsestoward one or more first objects, wherein a portion of each of theplurality of NIR light pulses is reflected off of the one or more firstobjects, receiving and focusing, using the imaging lens, the portion ofeach of the plurality of NIR light pulses reflected off of the one ormore first objects onto the image sensor, and detecting, using the imagesensor, a three-dimensional image of the one or more first objects bydetermining a time of flight for the portion of each of the plurality ofNIR light pulses from emission to detection. The imaging lens includesan aperture stop and a wavelength-selective filter positioned at theaperture stop. The wavelength-selective filter has a first region and asecond region surrounding the first region. The wavelength-selectivefilter is configured to transmit NIR light through the first region andthe second region, and to transmit visible light through the firstregion only. The method further includes performing computer visionusing the imaging system by: receiving and focusing, using the imaginglens, visible light from ambient light source reflected off of one ormore second objects onto the image sensor, and detecting, using theimage sensor, a two-dimensional intensity image of the one or moresecond objects.

According to a further embodiment of the present invention, an imagesensor for sensing light in a first wavelength range and a secondwavelength range includes a two-dimensional array of pixels and aprocessor. The processor is configured to measure light intensity foreach pixel of the array of pixels in the first wavelength range, andmeasure light intensities in the second wavelength range for a set ofpixel groups. Each pixel group includes m×n pixels of the array ofpixels, where m and n are integers, and at least one of m and n isgreater than one. In some embodiments, the first wavelength rangecorresponds to visible wavelengths, and the second wavelength rangecorresponds to near infrared (NIR) wavelengths. In some embodiments, mis equal to two, and n is equal to two. In some embodiments, measuringlight intensities in the second wavelength range for the set of pixelgroups includes reading out a total amount of charge for each group ofm×n pixels. In some alternative embodiments, measuring light intensitiesin the second wavelength range for the set of pixel groups includesreading out an amount of charge for each pixel of the array of pixels,and calculating a total amount of charge for each group of m×n pixels bysumming the amount of charge of the m×n pixels in each group.

Numerous benefits are achieved by way of the present invention overconventional techniques. For example, embodiments of the presentinvention provide an imaging lens that may be characterized by a lowerf-number for NIR light and a higher f-number for visible light byutilizing a wavelength-selective filter at its aperture stop. Moreover,embodiments of the present invention provide an image sensor that may beoperated at a lower resolution mode for NIR light using pixel binningand at a higher resolution mode for visible light using native pixelresolution. The imaging lens and the image sensor may be suitable foruse as a TOF depth sensor with active illumination in the NIR wavelengthrange where a faster lens and more light integration are desired, aswell as a computer vision sensor with passive illumination in thevisible wavelength range where higher image resolution and greater depthof field are desired. The imaging lens may be suitable for use for bothimaging visible light at a lower photo speed and imaging IR light at afaster photo speed. These and other embodiments of the invention alongwith many of its advantages and features are described in more detail inconjunction with the text below and attached figures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates schematically a system including an imaging systemaccording to an embodiment of the present invention.

FIG. 2 illustrates schematically an imaging system including an imaginglens and an image sensor according to an embodiment of the presentinvention.

FIG. 3 shows a schematic plan view of a wavelength-selective filter thatmay be used in an imaging lens according to an embodiment of the presentinvention.

FIG. 4A is a simplified plot illustrating a transmittance curve as afunction of wavelength for a central region of the wavelength-selectivefilter illustrated in FIG. 3, according to an embodiment of the presentinvention.

FIG. 4B is a simplified plot illustrating a transmittance curve as afunction of wavelength for an outer region of the wavelength-selectivefilter illustrated in FIG. 3, according to an embodiment of the presentinvention.

FIG. 5 illustrates a schematic cross-sectional view of awavelength-selective filter according to some embodiments of the presentinvention.

FIG. 6 illustrates a schematic imaging system according to someembodiments of the present invention.

FIG. 7 shows a ray tracing diagram of an exemplary imaging system for afield point (e.g., collimated rays at a certain incidence angle)according to some embodiments of the present invention.

FIG. 8 shows intensity distributions at the image sensor as simulated bythe ray tracing according to some embodiments of the present invention.

FIG. 9 illustrates a schematic cross-sectional diagram of awavelength-selective filter that may be used in an imaging systemaccording to some embodiments of the present invention.

FIG. 10A shows the intensity distribution of a ghost image from raytracing simulation with the wavelength-selective filter illustrated inFIG. 6 according to some embodiments of the present invention.

FIG. 10B shows the intensity distribution of a ghost image from raytracing simulation with the wavelength-selective filter illustrated inFIG. 9 according to some embodiments of the present invention.

FIG. 10C shows the ratio of the ghost image intensity using thewavelength-selective filter illustrated in FIG. 6 and the ghost imageintensity using the wavelength-selective filter illustrated in FIG. 9according to some embodiments of the present invention.

FIG. 11 illustrates a schematic cross-sectional diagram of awavelength-selective filter according to some other embodiments of thepresent invention.

FIG. 12 shows a transmittance curve and a reflectance curve of a “blackcoating,” as illustrated in FIG. 11, as a function of wavelengthaccording to some embodiments of the present invention.

FIG. 13 shows a reflectance curve of a second multilayer thin film, asillustrated in FIG. 11, as a function of wavelength according to someembodiments of the present invention.

FIG. 14 shows an exemplary quantum efficiency (Q.E.) curve as a functionof wavelength of an image sensor according to an embodiment of thepresent invention.

FIG. 15 illustrates schematically a plan view of an image sensoraccording to an embodiment of the present invention.

FIG. 16 illustrates schematically a mode of operating an image sensoraccording to an embodiment of the present invention.

FIG. 17 illustrates schematically an imaging system according to anotherembodiment of the present invention.

FIG. 18 illustrates schematically an imaging system according to afurther embodiment of the present invention.

FIG. 19 is a simplified flowchart illustrating a method of operating animaging system according to an embodiment of the present invention.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

The present invention relates generally to imaging systems with amultiple f-number lens. In optics, the f-number (sometimes referred toas the focal ratio, f-ratio, f-stop, or relative aperture) of a lens isthe ratio of the lens's focal length to the diameter of the entrancepupil. The f-number is a dimensionless number that is a quantitativemeasure of lens speed. Thus, the f-number or f/# is given by:

${{f/\#} = \frac{f}{D}},$where f is the focal length, and D is the diameter of the entrance pupil(effective aperture). A higher f-number implies a smaller diameter stopfor a given focal-length lens. Since a circular stop has area A=πr²,doubling the aperture diameter and therefore halving the f-number willadmit four times as much light into the system. Conversely, increasingthe f-number of an imaging lens decreases the amount of light entering acamera by decreasing the aperture size. For example, doubling thef-number will admit ¼ as much light into the system.

To maintain the same photographic exposure when doubling the f-number,the exposure time would need to be four times as long, or alternatively,the illumination would need to be increased to a level four times ashigh as the original level. Increasing the f-number may have the benefitof increasing the depth of field (DoF) and increasing the spatialresolution of an image (e.g., as measured by modulation transferfunction or MTF).

FIG. 1 illustrates schematically a system 100 that includes an imagingsystem 102 and an illumination source 104 according to an embodiment ofthe present invention. The system 100 may be integrated in a goggle, asillustrated in FIG. 1, that can be worn by a user for virtual reality(VR) or augmented reality (AR) experiences. The system 100 may includeother optical and electronic components for creating VR and ARexperiences.

In one embodiment, the imaging system 102 and the illumination source104 may be used for time-of-flight (TOF) depth sensing. The illuminationsource 104 can be configured to emit a plurality of laser pulses. Aportion of each of the plurality of laser pulses may be reflected off ofan object in front of the user. The portion of each of the plurality oflaser pulses reflected off of one or more objects may be received andimaged by the imaging system 102. The imaging system 102 can beconfigured to determine a time of flight for each of the laser pulsesfrom emission to detection, thereby determining the distance of theobject from the user. The illumination source 104 may comprise a lasersource, such as a vertical-cavity surface-emitting laser (VCSEL). Insome embodiments, the laser source may be configured to emit laserpulses in the near infrared (NIR) wavelength range, for example in thewavelength range from about 750 nm to about 1400 nm. The illuminationsource 104 may also include a collimation lens for collimating theplurality of laser pulses.

In some embodiments, the imaging system 102 may also be used forcomputer vision. When used for computer vision, the imaging system 102is configured to image objects in front of the user that are illuminatedby passive ambient light in the visible wavelength range. By using ashared imaging system for both TOF depth sensing and computer vision,lower cost and more compact system design may be realized. It should beunderstood that, although the imaging system 102 is described above aspart of an AR or VR system, the imaging system 102 may be used in othersystems. In other embodiments, the world cameras (WC) 106 and 108, aswell as the picture camera 110, may also be configured for dualfunctions, i.e., for imaging both visible and infrared light.

In some embodiments, the system 100 may operate the imaging system 102in a time-shared fashion such that depth sensing and computer vision arealternately performed at different time slots. In some embodiments, theduration of each time slot may range from about 1 ms to about 50 ms, sothat there is no significant latency in either depth sensing or computervision. In other embodiments, the system 100 may operate the imagingsystem 102 to perform depth sensing and computer vision simultaneously,as described in more detailed below.

FIG. 2 illustrates schematically an imaging system 200 that may be usedfor dual-wavelength sensing according to some embodiments of the presentinvention. For example, the imaging system 200 may be used for both TOFdepth sensing in the NIR wavelength range and computer vision in thevisible wavelength range. The imaging system 200 includes an imaginglens 210 and an image sensor 220 positioned at an image plane of theimaging lens 210. The imaging lens 210 may include one or more lenselements 216 a-216 e disposed along an optical axis. The imaging lensmay further include an aperture stop 212 that may define the entrancepupil size. In a lens system, the limiting diameter that determines theamount of light that reaches the image is called the aperture stop. Insome embodiments, the aperture stop may be positioned near the front ofa compound imaging lens. In some other embodiments, the aperture stopmay be positioned between two groups of lens elements of a compoundimaging lens (e.g., as illustrated in FIG. 2). In such cases, theentrance pupil size is determined by the image of the aperture stopformed by the lens elements preceding the aperture stop. In thefollowing, it is assumed that the entrance pupil size is the same as theaperture stop size.

When the imaging system 200 is used for TOF depth sensing, it may beadvantageous to configure the imaging lens 210 as a fast lens so that arelatively low power laser source may be used for active illumination.Lower power illumination may lead to lower cost, smaller form factor,and lower power consumption, among other advantages. In some cases, arelatively low f/#, for example in a range from about f/1 to aboutf/1.4, may be desirable for TOF depth sensing. In contrast, when theimaging system 200 is used for computer vision, it may be advantageousto configure the imaging lens 210 as a slow lens so that higher spatialresolution and greater depth of field (DoF) may be achieved. In somecases, a relatively high f/#, for example in a range from about f/2 toabout f/2.8, may be desirable for computer vision. The imaging system200 may be applied to other applications where it may be desirable tohave different lens speeds for sensing light in different wavelengthranges (e.g., for infrared sensing and visible light sensing).

According to an embodiment of the present invention, the imaging lens210 includes a filter 214 positioned at the aperture stop 212 that mayfunction as a wavelength selective filter. FIG. 3 shows a schematic planview of a filter 214 that may be used in the imaging lens 210 accordingto an embodiment of the present invention. The filter 214 may includetwo regions: a central (e.g., circular) region 310 with a first diameterD₁, and an outer (e.g., annular) region 320 surrounding the centralregion 310. The outer region 320 is characterized by a second diameterD₂ as its outer diameter. The second diameter D₂ may be substantiallythe same as the diameter of the aperture stop 212. It should beunderstood that, although the central region 310 is depicted as having acircular shape in FIG. 3, other shapes, such as elliptical, square,rectangular shapes can also be used. Similarly, although the outerregion 320 is depicted as having an annular shape in FIG. 3, othershapes are also possible.

FIG. 4A is a plot of an exemplary transmittance curve as a function ofwavelength for the central region 310 of the filter 214 according to anembodiment of the present invention. FIG. 4B is a plot of an exemplarytransmittance curve as a function of wavelength for the outer region 320of the filter 214 according to an embodiment of the present invention.As illustrated in FIG. 4A, the central region 310 of the filter 214 maybe configured to have a first transmission band 430 in the NIRwavelength range (e.g., from about 800 nm to about 950 nm) and a secondtransmission band 440 in the visible (VIS) wavelength range (e.g., fromabout 400 nm to about 700 nm). Accordingly, the central region 310 maybe characterized by high transmittance values in both the NIR and thevisible wavelength ranges. As illustrated in FIG. 4B, the outer region320 may be configured to have only one transmission band 450 in the NIRwavelength range (e.g., from about 800 nm to about 950 nm), such thatthe outer region 320 is characterized by high transmittance values inthe NIR wavelength range but low transmittance values in the visiblewavelength range.

In some embodiments, the filter 214 may comprise a multilayer thin filmstack formed on a surface of a transparent substrate such as glass. Amultilayer thin film may comprise a periodic layer system composed fromtwo or more materials of differing indices of refraction. This periodicsystem may be engineered to significantly enhance the transmittance ofthe surface in one or more desired wavelength ranges, while suppressingthe transmittance of the surface in other wavelength ranges. The maximumtransmittance may be increased up to nearly 100% with increasing numberof layers in the stack. The thicknesses of the layers making up themultilayer thin film stack are generally quarter-wave, designed suchthat transmitted beams constructively interfere with one another tomaximize transmission and minimize reflection. In one embodiment, themultilayer thin film stack in the central region 310 may be engineeredto have two high transmittance bands, one in the visible wavelengthrange and the other in the NIR wavelength range, and have lowtransmittance for all other wavelengths. The multilayer thin film stackin the annular region 320 may be engineered to have only one hightransmittance band in the NIR wavelength range, and have lowtransmittance for all other wavelengths. In other embodiments, othertypes of bandpass filters, such as metasurface filter, may be used.

FIG. 5 illustrates a schematic cross-sectional view of awavelength-selective filter 500 according to some embodiments of thepresent invention. The filter 500 may include a transparent substrate502 such as a piece of glass, a first multilayer thin film 510 disposedon a front surface of the substrate 502, and a second multilayer thinfilm 520 disposed on the first multilayer thin film 510. The firstmultilayer thin film 510 may have a circular shape with a diameter D₂.The second multilayer thin film 520 may have an annular shape with aninner diameter D₁ and an outer diameter D₂. In some embodiments, thefilter 500 may further include an anti-reflective coating 530 on theback surface of the substrate 502.

The first multilayer thin film 510 may be configured to have atransmittance curve that exhibits a first transmission band 430 in theNIR wavelength range (e.g., about 800 nm to about 950 nm) and a secondtransmission band 440 in the visible (VIS) wavelength range (e.g., about400 nm to about 700 nm), as illustrated in FIG. 4A. The secondmultilayer thin film 520 may be configured as a high-pass filter thattransmits light in the NIR wavelength range and blocks light in thevisible wavelength range, as illustrated by the dashed curve 460 in FIG.4A. As such, the combination of the first multilayer thin film 510 andthe second multilayer thin film 520 may result in an effectivetransmittance curve 450 as illustrated in FIG. 4B for the outer regionof the filter 500. Thus, the outer region of the filter 500 mayeffectively transmit only light in the NIR wavelength range, while thecentral region of the filter 500 may transmit light in both visible andNIR wavelength ranges.

When the filter 214 or 500 is positioned at the aperture stop 212 in theimaging lens 210 as illustrated in FIG. 2, the filter 214 or 500 mayeffectively give rise to two different apertures for the imaging lens210 depending on the wavelength range of the light being imaged.Referring to FIGS. 3 and 4A-4B, when the imaging lens 210 is used forimaging NIR light, for example for TOF depth sensing where theillumination laser source 104 (as illustrated in FIG. 1) operates in theNIR wavelength range, the NIR light is transmitted through both thecentral region 310 and the outer region 320 of the filter 214. Thus, theeffective aperture of the imaging lens 210 for NIR light is the seconddiameter D₂. When the imaging lens 210 is used for imaging visiblelight, for example for computer vision where the illumination is fromthe ambient visible light, the visible light is transmitted only throughthe central region 310. Thus, the effective aperture of the imaging lens210 for visible light is the first diameter D₁. The imaging lens 210with the wavelength-selective filter 214 may be applied to otherapplications where it may be desirable to have different lens speeds forsensing light in different wavelength ranges.

Assume that the imaging lens 210 has a focal length f. When the imaginglens is used for imaging visible light, the imaging lens 210 may becharacterized by a first f/# for visible light given by,

${f/\#_{VIS}} = {\frac{f}{D_{1}}.}$When the imaging lens is used for imaging NIR light, the imaging lens210 may be characterized by a second f/# for NIR light given by,

${f/\#_{NIR}} = {\frac{f}{D_{2}}.}$

Thus, the imaging lens 210 can be configured to have a relatively lowf/#_(NIR) for TOF depth sensing in the NIR wavelength range, and arelatively high f/#_(VIS) for computer vision in the visible wavelengthrange. For TOF depth sensing, a lower f/# means that more activeillumination NIR light can pass through the imaging lens 210. Thereforea relatively low power laser source may be used for illumination, whichmay lead to lower cost, smaller form factor, and lower powerconsumption, among other advantages. In some embodiments, the value ofD₂ may be chosen such that f/#_(NIR) is in a range from about f/1 toabout f/1.4.

For computer vision in the visible wavelength rage, a higher f/# mayafford higher spatial resolution at the image plane (e.g., as measuredby MTF) and greater DoF, among other advantages. In fact, a lower f/#may not be desired when imaging visible light in some cases. Asdescribed more fully below, image sensors typically have higher quantumefficiencies in the visible wavelength range than in the NIR wavelengthrange. Thus, the image sensor may be saturated when a fast lens is usedfor imaging visible light. In some embodiments, the value of D₁ may bechosen such that f/#_(VIS) is in a range from about f/2 to about f/2.8.The intensity ratio between VIS and NIR modes can be controlled bysetting the ratio D₁/D₂ accordingly. In some embodiments, a ratio ofD₁/D₂ may be chosen to be in the range from about 0.4 to about 0.6. Inone embodiment the ratio of D₁/D₂ may be chosen to be about 0.5, so thatthe value of f/#_(VIS) is about twice as large as the value off/#_(NIR).

FIG. 6 illustrates a schematic imaging system according to someembodiments. The imaging system may include a wavelength-selectivefilter 600, an optical lens 610, and an image sensor 620. Although asingle lens element is depicted for the optical lens 610 in FIG. 6 forsimplicity of illustration, the optical lens 610 may include severallens elements. The filter 600 may include a transparent substrate 602such as a piece of glass, a first multilayer thin film 604 that has acircular shape with a first diameter D₁, and a second multilayer thinfilm 606 that has an annular shape surrounding the first multilayer thinfilm 604 with an outer diameter of D₂. The first multilayer thin film604 may be configured to have high transmittance for both the visibleand NIR wavelength ranges, and the second multilayer thin film 606 maybe configured to have high transmittance for only the NIR wavelengthrange, as discussed above.

As illustrated in FIG. 6, an incoming light ray in the visiblewavelength range may be transmitted by the first multilayer thin film604 and form an image spot 622 at the image sensor, as illustrated bythe light path represented by the solid arrows. A portion of theincoming light may be reflected by the image sensor 620 and incident ona back side of the second multilayer film 606, as illustrated by thelight path represented by the dashed arrows. For incoming light in thevisible wavelength range, the reflected light may be reflected by thesecond multilayer thin film 606, as the second multilayer thin film 606is configured to have low transmittance values and high reflectancevalues in the visible wavelength range. The light reflected by thesecond multilayer thin film 606 may form a ghost image 624 at the imagesensor 620. Note that, for incoming light in the NIR wavelength range,the portion of the light reflected by the image sensor 620 and incidenton the back side of the second multilayer thin film 606 will be mostlytransmitted by the second multilayer thin film 606, as the secondmultilayer thin film 606 is configured to have high transmittance valuesin the NIR wavelength range. Thus, the filter 600 may not present asignificant ghost image problem for light in the NIR wavelength range.

FIG. 7 shows a ray tracing diagram of an exemplary imaging system for afield point (e.g., collimated rays at a certain incidence angle)according to some embodiments. The image system may include awavelength-selective filter 700, an optical lens 710, and an imagesensor 720. FIG. 8 shows intensity distributions at the image sensor 720as simulated by the ray tracing. As illustrated, the intensitydistributions show an image point 810, as well as a ghost image 820. Theghost image may obscure the real image. Therefore, it may be desirableto prevent the formation of the ghost image.

FIG. 9 illustrates a schematic cross-sectional diagram of awavelength-selective filter 900 that may be used in an imaging systemand may prevent ghost image formation according to some embodiments.Similar to the wavelength-selective filter 600 illustrated in FIG. 6,the filter 900 includes a transparent substrate 602, a first multilayerthin film 604 formed on a front side of the substrate 602 having acircular shape with a first diameter D₁, and a second multilayer thinfilm 606 formed on the front side of the substrate 602 having an annularshape surrounding the first multilayer thin film 604 with an outerdiameter of D₂. The first multilayer thin film 604 may be configured tohave high transmittance values in both the visible and NIR wavelengthranges, and the second multilayer thin film 606 may be configured tohave high transmittance values in only the NIR wavelength range, asdiscussed above.

The filter 900 may further include a third thin film 910 formed on aback side of the substrate 602. The third thin film 910 may have anannular shape with an outer diameter D₂ and an inner diameter D₃. D₃ maybe slightly greater than the inner diameter D₁ of the second multilayerthin film 606, so as not to block incoming light rays entering theimaging system through the central region (e.g., the first multilayerthin film 604) of the wavelength-selective filter 600. In someembodiments, the value of D₃ may depend on the thickness of thesubstrate 602. For a relatively thin substrate 602, D₃ may be comparableto D₁. The third thin film 910 may be configured to have high absorptioncoefficients in the visible wavelength range and high transmittancevalues in the NIR wavelength range. Thus, the third thin film 910 may bereferred to as a “black coating.” As visible light reflected off of theimage sensor 620 incident on the third thin film 910, a significantportion of it may be absorbed by the third thin film 910, and only asmall portion of it may be transmitted by the third thin film 910 andincident on the back surface of the second multilayer thin film 606 asillustrated by the light path represented by the thinner dashed arrowsin FIG. 9. Therefore, the intensity of the ghost image 624 may besignificantly reduced as compared to the case where the filter 600without the “black coating” is used as illustrated in FIG. 6.

FIG. 10A shows the intensity distribution of a ghost image from raytracing simulation using the wavelength-selective filter 600 illustratedin FIG. 6 according to some embodiments. FIG. 10B shows the intensitydistribution of a ghost image from ray tracing simulation using thewavelength-selective filter 900 illustrated in FIG. 9 that includes the“black coating” 910 according to some embodiments. As illustrated, theintensity of the ghost image may be significantly reduced by includingthe “black coating” 910 in the wavelength-selective filter 900. FIG. 10Cshows the ratio of the ghost image intensity using thewavelength-selective filter 600 that does not include a “black coating”and the ghost image intensity using the wavelength-selective filter 900with the “black coating” 910. As illustrated, the ghost image intensitycan be reduced by as much as 20 fold by including the “black coating”910 in the wavelength-selective filter 900.

FIG. 11 illustrates a schematic cross-sectional diagram of awavelength-selective filter 1100 according to some other embodiments.The filter 1100 may include a transparent substrate 1102, a firstmultilayer thin film 1110 formed on a front surface of the substrate1102. The first multilayer thin film 1110 may be configured to have afirst transmission band 430 in the NIR wavelength range and a secondtransmission band 440 in the visible wavelength range, as illustrated inFIG. 4A. The filter 1100 may further include a second multilayer thinfilm 1120 formed on the outer region of the first multilayer thin film1110. The second multilayer thin film 1120 may be configured to be ahigh-pass filter similar to the wavelength-selective filter 500illustrated in FIG. 5. The filter 1100 may further include ananti-reflective coating 1130 formed on a back surface of the substrate1102. The anti-reflective coating 1130 can prevent or reduce the amountof incoming light being reflected off of the back surface of thesubstrate 1102. The filter 1100 may further include a “black coating”1140 formed on the back surface of the anti-reflective coating 1130. The“black coating” 1140 may be configured to absorb visible light andtransmit NIR light as discussed above.

FIG. 12 shows a transmittance curve 1210 and a reflectance curve 1220 ofthe “black coating” 1140 as a function of wavelength according to someembodiments. A transmittance curve 1230 of the first multilayer thinfilm 1110 is also shown. As illustrated, the “black coating” 1140 can beconfigured to have low transmittance values for the visible wavelengthrange from about 400 nm to about 700 nm, and high transmittance valuesin the NIR wavelength range from about 800 nm to about 950 nm. The“black coating” 1140 may have relatively high reflectance values in thewavelength range from about 700 nm to about 800 nm, but this may notsignificantly affect the performance of the wavelength-selective filter1100 as light in this wavelength range is mostly blocked by the firstmultilayer thin film 1110 as evidenced by the transmittance curve 1230of the first multilayer thin film 1110.

Note that the “black coating” 1140 has both low reflectance values andlow transmittance values in the visible wavelength range. Thus, the“black coating” 1140 may substantially absorb visible light, therebypreventing visible light reflected off of the image sensor 620 (asillustrated in FIG. 9) from being transmitted and incident on the backside of the second multilayer thin film 606 to form a ghost image 624 onthe image sensor 620. In contrast, the anti-reflective coating 1130 isnormally configured to have low reflectance values but hightransmittance values. Thus, visible light reflected off of the imagesensor 620 may be transmitted by the anti-reflective coating 1130 and bereflected by the second multilayer thin film 1120 to form the ghostimage 624 on the image sensor 620 in absence of the “black coating”1140.

FIG. 13 shows a reflectance curve 1310 of the second multilayer thinfilm 1120 as a function of wavelength according to some embodiments. Asillustrated, the second multilayer thin film 1120 may be configured tohave low reflectance values (thus high transmittance values) only in theNIR wavelength range from about 800 nm to about 950 nm, and relativelyhigh reflectance values for all other wavelengths. FIG. 13 also showsthe reflectance curve 1220 of the “black coating” 1140, as well as thetransmittance curve 1230 of the first multilayer thin film 1110. Asillustrated, the low reflectance values of the “black coating” 1140 inthe visible wavelength range may reduce reflection of light in thevisible wavelength range, thereby reduce the intensity of the ghostimage.

FIG. 14 shows an exemplary quantum efficiency (Q.E.) curve 1410 as afunction of wavelength of an image sensor 220 that may be used in theimaging system 200 as illustrated in FIG. 2, according to an embodimentof the present invention. As illustrated, the quantum efficiency of theimage sensor 220 in the visible (VIS) wavelength range can be as much asfour times of the quantum efficiency in the NIR wavelength range.Therefore, a low f/# lens may allow too much visible light to passthrough the imaging lens 210 to the image sensor 220 and may saturatethe image sensor 220.

In some embodiments, the image sensor 220 in the imaging system 200illustrated in FIG. 2 may comprise a charge-coupled device (CCD) or acomplementary metal-oxide semiconductor (CMOS) device that convertslight into electrons in a two-dimensional array of pixel cells. FIG. 15illustrates schematically a plan view of the image sensor 220 accordingto an embodiment of the present invention. The image sensor 220 mayinclude a two-dimensional array of pixel cells 222. The value of theaccumulated charge of each pixel cell 222 may be read out to obtain anintensity distribution of the image. When the imaging system 200 is usedfor computer vision in the visible wavelength range, it may be desirableto have the highest possible spatial resolution at the image sensor 220.On the other hand, when the imaging system 200 is used for TOF depthsensing in the NIR wavelength range, it may be advantageous to have morelight integration at the expense of pixel resolution to achieve bettersignal to noise ratio (SNR).

According to some embodiments of the present invention, the image sensor220 may be operated at different resolution modes for the visiblewavelength range and the NIR wavelength range. In one embodiment, theimage sensor 220 may be operated at the native resolution for thevisible wavelength range, i.e., at the maximum possible resolution thatthe physical pixel size of the image sensor can support. Thus, forcomputer vision in the visible wavelength range, the image sensor 220may be operated such that the accumulated charge in each pixel cell 222is read out.

For the NIR wavelength range, the image sensor 220 may be operated at aresolution that is lower than the native resolution for greater lightintegration. FIG. 16 illustrates schematically a mode of operating theimage sensor 220 according to an embodiment of the present invention.The two-dimensional array of pixel cells 222 may be binned into 2×2groups 224. Each group 224 includes four pixel cells 222 a-222 d. Thismode of operation can be referred to as image sensor pixel binning. Inother embodiments, other binning configurations may be used. Forexample, the pixel cells 222 of the image sensor 220 may be binned inton×n groups, where n is an integer greater than one. The pixels of theimage sensor may also be binned into m×n groups, where m and n areintegers and at least one of m and n is greater than one, and m may ormay not be equal to n. By binning the pixels, the spatial resolution maybe reduced as compared to the native resolution. When the image sensor220 is used in an imaging system that includes the wavelength-selectivefilter 214, 500, 600, 900, or 1100, since the spatial resolution of theimaging system (e.g., as measured by modulation transfer function orMTF) may be lower in the NIR wavelength range because of the greatereffective aperture size, the reduction of spatial resolution at theimage sensor may not be detrimental. With the greater light integrationafforded by binning, a relatively low power laser source may be used foractive illumination. Lower power illumination may lead to lower cost,smaller form factor, and lower power consumption, among otheradvantages.

In one embodiment, binning may be performed at the analog level, wherethe value of the total accumulated charge for the m×n pixels in eachgroup is read out. In such cases, the readout noise is not added. Inanother embodiment, binning may be performed at the digital level, wherethe value of the accumulated charge for each pixel is read out, and thereadout values for the m×n pixels in each group are then summed. In suchcases, the readout noise is added in the summation process. Thus, thelater embodiment may be more appropriate where the readout noise isrelatively low.

As described above, the imaging system 200 illustrated in FIG. 2includes an imaging lens 210 that may be characterized by a lowerf-number for NIR light and a higher f-number for visible light byutilizing a wavelength-selective filter 214 at its aperture stop, and animage sensor 220 that may be operated at a lower resolution mode for NIRlight using pixel binning and at a higher resolution mode for visiblelight. The imaging system 200 may be suitable for use as a TOF depthsensor with active illumination in the NIR wavelength range where afaster lens and more light integration are desired, as well as acomputer vision sensor with passive illumination in the visiblewavelength range where higher image resolution and greater depth offield are desired.

FIG. 17 is a schematic diagram illustrating an imaging system 1700according to another embodiment of the present invention. The imagingsystem 1700 may include a plurality of lens elements 1702 a-1702 f, anda filter 214 positioned at the aperture stop 212. The imaging system 800may further include a dichroic beam splitter 1710 positioned in theoptical path after the filter 214. The dichroic beam splitter 1710 maybe configured to transmit visible light along a first optical path, andreflect IR light along a second optical path. The imaging system 1700may further include a first image sensor 1720 (VIS sensor) for visiblelight, and a second image sensor 1730 (IR sensor) for IR light. Thefirst image sensor 1720 is disposed along the first optical path andconfigured to receive the visible light transmitted by the dichroic beamsplitter 1710. The second image sensor 1730 is disposed along the secondoptical path and configured to receive the IR light reflected by thedichroic beam splitter 1710. In this fashion, visible light and IR lightmay be imaged by the first image sensor 1720 and the second image sensor17830, respectively, at the same time. In this configuration, the firstoptical path to the first image sensor 1720 and the second optical pathto the second image sensor 1730 are perpendicular to each other.

FIG. 18 is a schematic diagram illustrating an imaging system 1800according to yet another embodiment of the present invention. Theimaging system 1800 is similar to the imaging system 1700 in that italso includes a dichroic beam splitter 1710 positioned after the filter214, and configured to transmit visible light along a first optical pathand to reflect IR light along a second optical path. The imaging system1800 further includes a mirror 1810 positioned along the second opticalpath and configured to reflect IR light toward the second image sensor1730. In this configuration, the first optical path to the first imagesensor 1720 and the second optical path to the second image sensor 1730are parallel to each other. The imaging system 1800 may further includea lens element 1820 positioned after the mirror 1810 along the secondoptical path for refocusing IR light at the second image sensor 1730.

FIG. 19 is a simplified flowchart illustrating a method 1900 ofoperating an imaging system according to an embodiment of the presentinvention. The method 1900 includes performing three-dimensional sensingusing the imaging system. In some embodiments, performing thethree-dimensional sensing is performed in a first time slot. The imagingsystem may include a near infrared (NIR) light source, an imaging lens,and an image sensor positioned at an image plane of the imaging lens.

In an embodiment, three-dimensional sensing may be performed by:emitting, using the NIR light source, a plurality of NIR light pulsestoward one or more first objects (1910). A portion of each of theplurality of NIR light pulses may be reflected off of the one or morefirst objects. The method also includes receiving and focusing, usingthe imaging lens, the portion of each of the plurality of NIR lightpulses reflected off of the one or more first objects onto the imagesensor (1912). The imaging lens may include an aperture stop and awavelength-selective filter positioned at the aperture stop. Thewavelength-selective filter may have a first region and a second regionsurrounding the first region. In one embodiment, thewavelength-selective filter is configured to transmit NIR light throughboth the first region and the second region, and to transmit visiblelight through the first region only. The method further includesdetecting, using the image sensor, a three-dimensional image of the oneor more first objects by determining a time of flight for the portion ofeach of the plurality of NIR light pulses from emission to detection(1914).

The method 1900 further includes performing computer vision in a secondtime slot using the imaging system. Performing computer vision may beperformed in a second time slot following the first time slot. In anembodiment, computer vision may be performed by receiving and focusing,using the imaging lens, visible light from an ambient light sourcereflected off of one or more second objects onto the image sensor(1916), and detecting, using the image sensor, a two-dimensionalintensity image of the one or more second objects (1918). In someembodiments, some of the second objects can be the same as some of thefirst objects that were imaged in steps 1910-1914 described above.

According to an embodiment of the present invention, the image sensorincludes a two dimensional array of pixels. In some embodiments,detecting the three-dimensional image of the one or more first objectsis performed by reading out a total amount of charge for each group ofm×n pixels, where m and n are integers, and at least one of m and n isgreater than one. In some other embodiments, detecting thethree-dimensional image of the one or more first objects is performed byreading out an amount of charge for each pixel of the two-dimensionalarray of pixels, and calculating a total amount of charge for each groupof m×n pixels by summing the amount of charge of the m×n pixels in eachgroup, where m and n are integers, and at least one of m and n isgreater than one.

In one embodiment, detecting the two-dimensional intensity image of theone or more second objects is performed by reading out an amount ofcharge for each pixel of the two-dimensional array of pixels.

In some embodiments, the method 1900 may include alternately performingthree-dimensional sensing and computer vision in sequential time slots,and the duration of each time slot may range from about 1 ms to about 50ms.

In some other embodiments, the method 1900 may include performingthree-dimensional sensing and computer vision simultaneously using animaging system such as that illustrated in FIG. 17 or FIG. 18.

It should be appreciated that the specific steps illustrated in FIG. 19provide a particular method of 1900 according to an embodiment of thepresent invention. Other sequences of steps may also be performedaccording to alternative embodiments. For example, alternativeembodiments of the present invention may perform the steps outlinedabove in a different order. Moreover, the individual steps illustratedin FIG. 19 may include multiple sub-steps that may be performed invarious sequences as appropriate to the individual step. Furthermore,additional steps may be added or removed depending on the particularapplications. One of ordinary skill in the art would recognize manyvariations, modifications, and alternatives.

It is also understood that the examples and embodiments described hereinare for illustrative purposes only and that various modifications orchanges in light thereof will be suggested to persons skilled in the artand are to be included within the spirit and purview of this applicationand scope of the appended claims.

What is claimed is:
 1. An imaging system comprising: a near infrared(NIR) light source configured to emit a plurality of NIR light pulsestoward one or more first objects, wherein a portion of each of theplurality of NIR light pulses is reflected off of the one or more firstobjects; one or more lens elements configured to receive and focus theportion of each of the plurality of NIR light pulses reflected off ofthe one or more first objects onto an image plane, and to receive andfocus visible light reflected off of one or more second objects onto theimage plane; an aperture stop; a filter positioned at the aperture stop,the filter including: a central region with a first linear dimension,the central region being characterized by higher transmittance values inone or more wavelength ranges than in other wavelength ranges, whereinthe one or more wavelength ranges include an NIR wavelength range and avisible wavelength range; and an outer region surrounding the centralregion with a second linear dimension greater than the first lineardimension, the outer region being characterized by higher transmittancevalues in the NIR wavelength range than in the visible wavelength range;and an image sensor positioned at the image plane, the image sensorincluding a two-dimensional array of pixels, wherein the image sensor isconfigured to: detect a two-dimensional intensity image of the one ormore second objects in an unbinned pixel mode, wherein thetwo-dimensional intensity image is formed by light in the visiblewavelength range transmitted through only the central region of thefilter; and detect a time-of-flight three-dimensional image of the oneor more first objects in a binned pixel mode in which each respectivegroup of two or more adjacent pixels are binned as a binned pixel,wherein the time-of-flight three-dimensional image is formed by light inthe NIR wavelength range transmitted through both the central region andthe outer region of the filter.
 2. The imaging system of claim 1 whereinthe central region has a circular shape, and the outer region has aannular shape, and wherein the first linear dimension is a diameter ofthe central region, and the second linear dimension is an outer diameterof the outer region.
 3. The imaging system of claim 1 wherein a ratio ofthe first linear dimension and the second linear dimension ranges from0.4 to 0.6.
 4. The imaging system of claim 2 wherein a ratio of thefirst linear dimension and the second linear dimension is 0.5.
 5. Theimaging system of claim 1 wherein the one or more lens elements, theaperature stop, and the filter form an optical lens characterized by afirst f-number for light in the NIR wavelength range based on the secondlinear dimension, and by a second f-number for light in the visiblewavelength range based on the first linear dimension.
 6. The imagingsystem of claim 5 wherein the first f-number ranges from 1.0 to 1.4, andthe second f-number ranges from 2.0 to 2.8.
 7. The imaging system ofclaim 1 wherein, in the binned pixel mode, the respective group of twoor more adjacent pixels comprises m×n pixels, m and n being positiveintegers, and the image sensor reads out accumulated charge of therespective group of m×n pixels.
 8. An imaging system comprising: one ormore lens elements configured to receive and focus light in a nearinfrared (NIR) wavelength range reflected off of one or more firstobjects onto an image plane, and to receive and focus light in a visiblewavelength range reflected off of one or more second objects onto theimage plane; an aperture stop; a filter positioned at the aperture stop,the filter including: a central region with a first linear dimension,the central region being characterized by a first transmission band inthe NIR wavelength range and a second transmission band in the visiblewavelength range, the second transmission band not overlapping with thefirst transmission band; and an outer region surrounding the centralregion with a second linear dimension greater than the first lineardimension, the outer region being characterized by only one transmissionband in the NIR wavelength range; and an image sensor including atwo-dimensional array of pixels, wherein the image sensor is configuredto: detect a two-dimensional intensity image of the one or more secondobjects by reading out an amount of charge for each pixel of thetwo-dimensional array of pixels, wherein the two-dimensional intensityimage is formed by light in the visible wavelength range transmittedthrough only the central region of the filter; and detect athree-dimensional image of the one or more first objects by reading outa total amount of charge for each group of m×n pixels, m and n beingpositive integers, and at least one of m and n being greater than one,wherein the three-dimensional image is formed by light in the NIRwavelength range transmitted through both the central region and theouter region of the filter.
 9. The imaging system of claim 8 wherein thecentral region has a circular shape, and the outer region has a annularshape, and wherein the first linear dimension is a diameter of thecentral region, and the second linear dimension is an outer diameter ofthe outer region.
 10. The imaging system of claim 9 wherein the filterfurther includes: a thin film having an annular shape formed on a backsurface thereof, wherein the thin film is configured to absorb light inthe visible wavelength range and to transmit light in the NIR wavelengthrange.
 11. The imaging system of claim 8 wherein a ratio of the firstlinear dimension and the second linear dimension ranges from 0.4 to 0.6.12. The imaging system of claim 8 is characterized by a first f-numberfor light in the visible wavelength range that ranges from 1.0 to 1.4,and by a second f-number for light in the NIR wavelength range thatranges from 2.0 to 2.8.
 13. A method of operating an imaging system, theimaging system comprising a near infrared (NIR) light source, an imaginglens, and an image sensor comprising a two-dimensional array of pixelsand positioned at an image plane of the imaging lens, the methodcomprising: performing three-dimensional sensing using the imagingsystem by: emitting, using the NIR light source, a plurality of NIRlight pulses toward one or more first objects, wherein a portion of eachof the plurality of NIR light pulses is reflected off of the one or morefirst objects; receiving and focusing, using the imaging lens, theportion of each of the plurality of NIR light pulses reflected off ofthe one or more first objects onto the image sensor, wherein the imaginglens includes an aperture stop and a wavelength-selective filterpositioned at the aperture stop, the wavelength-selective filter havinga first region and a second region surrounding the first region, thewavelength-selective filter configured to transmit light in an NIRwavelength range through both the first region and the second region,and to transmit light in a visible light wavelength range through thefirst region only; and detecting, using the image sensor in a binnedpixel mode, a three-dimensional image of the one or more first objectsby binning a plurality of pixels of the two-dimensional array of pixelsand determining a time of flight for the portion of each of theplurality of NIR light pulses from emission to detection, wherein thethree-dimensional image is formed by light in the NIR wavelength rangetransmitted through both the first region and the second region of thewavelength-selective filter; and performing computer vision using theimaging system by: receiving and focusing, using the imaging lens,visible light from an ambient light source reflected or scattered off ofone or more second objects onto the image sensor, wherein the visiblelight is transmitted only through the first region of thewavelength-selective filter of the imaging lens; and detecting, usingthe image sensor in an unbinned pixel mode, a two-dimensional intensityimage of the one or more second objects, wherein the two-dimensionalintensity image is formed by light in a visible wavelength rangetransmitted through only the first region of the wavelength-selectivefilter.
 14. The method of claim 13 wherein performing three-dimensionalsensing is performed in a first time slot and performing computer visionis performed in a second time slot.
 15. The method of claim 14 wherein aduration of each of the first time slot and the second time slot is in arange from 1 ms to 50 ms.
 16. The method of claim 13 wherein the firstregion has a circular shape characterized by a first diameter, and thesecond region has a annular shape characterized by an outer seconddiameter greater than the first diameter.
 17. The method of claim 16wherein a ratio of the first diameter and the second diameter rangesfrom 0.4 to 0.6.
 18. The method of claim 13 wherein the binned pixelmode comprises reading out a total amount of charge for each group ofm×n pixels, m and n being positive integers, and at least one of m and nbeing greater than one.
 19. The method of claim 13 wherein the binnedpixel mode comprises: reading out an amount of charge for each pixel ofthe two-dimensional array of pixels; and calculating a total amount ofcharge for each group of m×n pixels by summing the amount of charge ofthe m×n pixels in each group, m and n being integers, and at least oneof m and n being greater than one.
 20. The method of claim 13 whereinthe unbinned pixel mode comprises reading out an amount of charge foreach pixel of the two-dimensional array of pixels and defining a pixelintensity for each pixel of the two-dimensional array of pixels based onthe amount of charge for each pixel of the two-dimensional array ofpixels.